Skip to content

fix lancedb out for recall after pipeline#1510

Merged
jdye64 merged 9 commits intoNVIDIA:mainfrom
jperez999:fix-lance-slim
Mar 10, 2026
Merged

fix lancedb out for recall after pipeline#1510
jdye64 merged 9 commits intoNVIDIA:mainfrom
jperez999:fix-lance-slim

Conversation

@jperez999
Copy link
Collaborator

Description

works on fixing the lancedb search and recall steps in the batch example

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.
  • If adjusting docker-compose.yaml environment variables have you ensured those are mimicked in the Helm values.yaml file.

@jperez999 jperez999 self-assigned this Mar 7, 2026
@jperez999 jperez999 requested a review from a team as a code owner March 7, 2026 02:53
@jperez999 jperez999 requested a review from jioffe502 March 7, 2026 02:53

schema = lancedb_schema(2048)
empty = pa.table({f.name: [] for f in schema}, schema=schema)
schema = pa.schema(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is something different about this than what is in the new central lancedb_schema function?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have a ton of places where we reference lancedb. We need to consolidate. I will change for that call to be used in all functions.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, agreed. Some of the functions that had been centralized seem to have been re-introduced. Lets make some time to tidy this up soon.

@jdye64
Copy link
Collaborator

jdye64 commented Mar 10, 2026

Unrelated to your changes but upstream changes that were needed previously now cause errors with this PR when running. Can you remove this snippet in batch_pipeline.py? It is around line 734

if hasattr(ingestor, "_create_lancedb_index"): ingestor._create_lancedb_index()

@jdye64 jdye64 merged commit 2039f9d into NVIDIA:main Mar 10, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants